Lens module array, image sensing device and fusing method for digital zoomed images

ABSTRACT

A lens module array for being assembled to a portable device is provided. The lens module array includes a wide-angle mono lens module, a narrow-angle mono lens module, and two color lens modules. While the wide-angle mono lens module, the narrow-angle mono lens module, and the two color lens modules are assembled onto the portable device, the wide-angle mono lens module, the narrow-angle mono lens module, and the two color lens modules are located at four vertices of a quadrangle, respectively. Besides, the two color lens modules are located at two opposite vertices.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefits of U.S. provisionalapplication Ser. No. 62/151,435, filed on Apr. 23, 2015 and Taiwanapplication serial no. 104124309, filed on Jul. 28, 2015. The entiretyof each of the above-mentioned patent applications is herebyincorporated by reference herein and made a part of this specification.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to a lens module array and particularly relates toa lens module array with four lens modules, an image sensing device, anda fusing method for a digital zoomed image.

2. Description of Related Art

With the development of technology, various smart electronic devices,such as tablet computers and smart phones, have become indispensabletools for the modern people. In particular, high-end smart electronicdevices are equipped with camera lenses that can generate high-qualityimages which are comparable to those generated by the general digitalcameras or may even replace them. The camera lenses of some high-endsmart electronic devices can challenge Digital SLRs in pixel number andimage quality, and some even have dual lenses for capturingthree-dimensional images.

For the electronic device with dual lenses, it is a common applicationto use the wide-angle lens (one of the dual lenses) to capture awide-angle image and use the telephoto lens (the other lens) to capturea narrow-angle image, and then select the wide-angle image or thenarrow-angle image as the target image according to the magnificationfor digitally magnifying one single target image to simulate the opticalzoom function. However, details of the image may not be retained due tothe digital magnification. The image would be blurred as themagnification increases. In the process of zooming the image, if thetarget image needs to be switched to the wide-angle image or thenarrow-angle image, the display of the image may not be smooth or jump.

SUMMARY OF THE INVENTION

The invention provides a lens module array, an image sensing device, anda fusing method for a digital zoomed image to cope with the problem thatthe conventional dual lens module is unable to satisfy both therequirements of image quality and compact size.

The lens module array of the invention is for being assembled to aportable device. The lens module array includes a wide-angle mono lensmodule, a narrow-angle mono lens module, and two color lens modules.When the wide-angle mono lens module, the narrow-angle mono lens module,and the two color lens modules are assembled onto the portable device,the wide-angle mono lens module, the narrow-angle mono lens module, andthe two color lens modules are located at four vertices of a quadranglerespectively. The two color lens modules are located at two oppositevertices.

The image sensing device of the invention includes an image processingmodule, a wide-angle mono lens module, a narrow-angle mono lens module,and two color lens modules. When the wide-angle mono lens module, thenarrow-angle mono lens module, and the two color lens modules areassembled to a portable device, the wide-angle mono lens module, thenarrow-angle mono lens module, and the two color lens modules arelocated at four vertices of a quadrangle respectively. The two colorlens modules are located at two opposite vertices. The image processingmodule digitally zooms images captured by the wide-angle mono lensmodule, the narrow-angle mono lens module, and the two color lensmodules and fuses the images into a composite image.

The fusing method of a digital zoomed image of the invention includesthe following steps. Images are captured by a wide-angle mono lensmodule, a narrow-angle mono lens module, and two color lens modulesrespectively. The wide-angle mono lens module, the narrow-angle monolens module, and the two color lens modules are located at four verticesof a quadrangle respectively. The two color lens modules are located attwo opposite vertices. The images captured by the wide-angle mono lensmodule, the narrow-angle mono lens module, and the two color lensmodules are processed into a composite image.

In an embodiment of the fusing method of the invention, a method ofprocessing the images into the composite image includes the followingsteps. Mono images captured by the wide-angle mono lens module and thenarrow-angle mono lens module are fused into a digital zoomed mono imageaccording to a zoom ratio. The digital zoomed mono image and colorimages captured by the color lens modules are fused into the compositeimage that conforms to the zoom ratio and is colorful.

In an embodiment of the fusing method of the invention, a method offusing the digital zoomed mono image and the color images captured bythe color lens modules includes the following steps. Pixel alignment isperformed on the digital zoomed mono image and the color images capturedby the color lens modules respectively. Utilizing the digital zoomedmono image as a basis, color information in the color images captured bythe color lens modules is filled into corresponding pixels atcorresponding positions in the digital zoomed mono image to obtain thecomposite image.

In an embodiment of the invention, an angle of view of the wide-anglemono lens module is in a range of 70-80 degrees and an angle of view ofthe narrow-angle mono lens module is in a range of 35-40 degrees.

In an embodiment of the invention, an angle of view of the two colorlens modules is in a range of 70-80 degrees.

In an embodiment of the invention, the quadrangle is square,rectangular, rhombic, or kite-shaped.

In an embodiment of the invention, a total number of pixels of thewide-angle mono lens module is the same as a total number of pixels ofthe color lens modules.

In an embodiment of the invention, the image processing module includesa storage unit and a processing unit, and the storage unit is coupled tothe processing unit.

In an embodiment of the invention, the processing unit includes an imageinputting module, an image pre-processing module, a feature pointanalysis module, an image zooming and warping module, an image fusingmodule, and a coloring module.

Based on the above, the lens module array, the image sensing device, andthe fusing method of a digital zoomed image of the invention utilize thewide-angle mono lens module and the narrow-angle mono lens module. Thus,under the condition that the area of the sensing element remains thesame, the invention is able to simulate the narrow-angle optical zoomingeffect and avoid increasing the overall thickness.

To make the aforementioned and other features and advantages of theinvention more comprehensible, several embodiments accompanied withdrawings are described in detail as follows.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the invention, and are incorporated in and constitute apart of this specification. The drawings illustrate exemplaryembodiments of the invention and, together with the description, serveto explain the principles of the invention.

FIG. 1 is a schematic diagram illustrating a lens module array assembledto a portable device according to an embodiment of the invention.

FIG. 2 is a flowchart illustrating a fusing method for a digital zoomedimage according to an embodiment of the invention.

DESCRIPTION OF THE EMBODIMENTS

Several embodiments of the invention are described in detail hereinafterwith reference to the figures. Regarding the reference numeralsmentioned in the following description, the same reference numerals indifferent figures are deemed to represent the same or similarcomponents. These embodiments are only part of the invention. Not allpossible embodiments are disclosed here in this specification. Moreprecisely, these embodiments are merely examples of the device/methoddefined by the scope of the invention.

FIG. 1 is a schematic diagram illustrating a lens module array assembledto a portable device according to an embodiment of the invention. Itshould be noted that the diagram is provided to facilitate explanation,not to limit the invention. The lens module array is presented by thephysical positions of four lens modules while the other components areillustrated simply by functional blocks.

With reference to FIG. 1, a lens module array 100 of this embodiment isassembled to a portable device 50. In this embodiment, the portabledevice 50 is an electronic device, such as a smart phone, a tabletcomputer, a personal digital assistant, and a head mounted display,which has a digital camera, a SLR camera, a digital video camera, awebcam, or other image capturing functions, for example. Nevertheless,the invention is not limited to the foregoing.

The lens module array 100 includes a wide-angle mono lens module 110, anarrow-angle mono lens module 120, and two color lens modules 130A and130B. When the wide-angle mono lens module 110, the narrow-angle monolens module 120, and the two color lens modules 130A and 130B areassembled onto the portable device 50, the wide-angle mono lens module110, the narrow-angle mono lens module 120, and the two color lensmodules 130A and 130B are located at four vertices S12 of a quadrangleS10 respectively. Besides, the two color lens modules 130A and 130B arelocated at two opposite vertices S12. An angle of view of the wide-anglemono lens module 110 is larger than an angle of view of the narrow-anglemono lens module 120. From another aspect, the narrow-angle mono lensmodule 120 may also be called a telephoto lens module.

All pixels of the wide-angle mono lens module 110 are used for sensingand recording the total luminance of the light that enters the pixel.All pixels of the narrow-angle mono lens module 120 are also used forsensing and recording the total luminance of the light that enters thepixel. In an embodiment, the wide-angle mono lens module 110 and thenarrow-angle mono lens module 120 may be black and white lens modules,which record the light intensity regardless of the wavelength range. Onthe other hand, all pixels of the color lens modules 130A and 130Binclude pixels for sensing and recording the intensity of light invarious wavelength ranges. For example, some pixels are used for sensingand recording the intensity of red light, some are used for sensing andrecording the intensity of green light, and some are used for sensingand recording the intensity of blue light. Of course, the light sensedand recorded by the pixels of the color lens modules 130A and 130B isnot limited to the red light, green light, and blue light, and may alsobe other light combinations as long as full-color image recording isachieved.

In comparison with the conventional dual lens module that uses awide-angle color lens module and a narrow-angle color lens module, thewide-angle mono lens module 110 and the narrow-angle mono lens module120 of this embodiment achieve higher resolution because the pixels donot need to be grouped to sense lights of different colors. On the otherhand, the mono lens module does not have a color filter which is used inthe color lens module and thus has a higher light absorbency. Therefore,even in a low-light environment, noise generation may be suppressedeffectively and the signal to noise ratio may be improved to generate aclear image. Hence, given that the area of the photosensitive elementremains the same, the narrow-angle mono lens module 120 that is thinnerin size may provide the same resolution as the conventional narrow-anglecolor lens module. Thereby, the overall thickness of the lens modulearray 100 of this embodiment is reduced.

Some aspects of the lens module array 100 of this embodiment areexplained below. However, it should be noted that the invention is notlimited thereto. In this embodiment, the angle of view of the wide-anglemono lens module 110 is in a range of 70-80 degrees and the angle ofview of the narrow-angle mono lens module 120 is in a range of 35-40degrees. The angle of view refers to an angle included between theleftmost side and the rightmost side of the image that may be capturedby the lens module in the horizontal direction. In the case of the sametotal number of pixels, the angle of view of the image captured by thewide-angle mono lens module 110 is larger and thus a wider image iscaptured, and the angle of view of the image captured by thenarrow-angle mono lens module 120 is smaller and thus more image detailsare retained. By processing the images captured by the wide-angle monolens module 110 and the narrow-angle mono lens module 120, a digitalzoom function that simulates optical zoom may be provided. Imageintegration will be described in brief later. In addition, the two colorlens modules 130A and 130B are disposed to impart color information tothe image obtained through integration. To provide the color informationfor the widest image, the angle of view of the color lens modules 130Aand 130B may be set the same as the angle of view of the wide-angle monolens module 110, e.g. in the range of 70-80 degrees. Moreover, the totalnumber of pixels of the color lens modules 130A and 130B may be the sameas that of the wide-angle mono lens module 110. To facilitateintegration of the images of the four lens modules, the quadrangle maybe square, rectangular, rhombic, or kite-shaped, for example.

The wide-angle mono lens module 110, the narrow-angle mono lens module120, and the color lens modules 130A and 130B may include aphotosensitive element for sensing the light intensity to generate animage respectively. The photosensitive element is a charge coupleddevice (CCD) or a complementary metal-oxide semiconductor (CMOS) device,for example. However, it should be noted that the invention is notlimited thereto. The wide-angle mono lens module 110, the narrow-anglemono lens module 120, and the color lens modules 130A and 130B are usedto capture images of a subject, which may be a specific object or ascene.

With reference to FIG. 1 again, the portable device 50 includes an imageprocessing module 51. The image processing module 51 processes theimages captured by the wide-angle mono lens module 110, the narrow-anglemono lens module 120, and the color lens modules 130A and 130B into acomposite image. In this embodiment, the image processing module 51includes a storage unit 52 and a processing unit 54. The storage unit 52is a fixed or a movable random access memory (RAM) in any form, aread-only memory (ROM), a flash memory, a hard disc, other similardevices, or a combination of the foregoing, for example. The storageunit 52 is coupled to the lens module array 100 for storing the imagescaptured by the wide-angle mono lens module 110, the narrow-angle monolens module 120, and the color lens modules 130A and 130B.

The processing unit 54 is a central processing unit (CPU), aprogrammable microprocessor for general or special use, a digital signalprocessor (DSP), a programmable controller, an application specificintegrated circuit (ASIC), a programmable logic device (PLD), othersimilar devices, or a combination of these devices, for example.

The storage unit 52 is coupled to the lens module array 100 and theprocessing unit 54 and obtains complete depth information of the subjectthrough the images captured by the wide-angle mono lens module 110 andthe narrow-angle mono lens module 120. The processing unit 54 forexample includes an image inputting module 541, an image pre-processingmodule 542, a feature point analysis module 543, an image zooming andwarping module 544, an image fusing module 545, and a coloring module546, which may be loaded to the storage unit 52 to perform the digitalzoom function. Steps of a digital zooming method executed by theportable device 50 are explained in detail in the following embodiment.

FIG. 2 is a flowchart illustrating a fusing method for a digital zoomedimage according to an embodiment of the invention. Here, the digitalzoomed image is fused using the electronic device of FIG. 1 as anexample. With reference to FIG. 1 and FIG. 2, first, the storage unit 52of the portable device 50 controls the wide-angle mono lens module 110,the narrow-angle mono lens module 120, and the color lens modules 130Aand 130B of the lens module array 100 to capture images of a scenerespectively, so as to generate four images to be inputted to the imageinputting module 541 (Step S110). In this embodiment, a main mono imagecaptured by the wide-angle mono lens module 110 has lower image qualitybut covers a larger viewing angle range. An auxiliary mono imagecaptured by the narrow-angle mono lens module 120 has higher imagequality but covers a smaller viewing angle range, and is used to assistthe digital zooming in the subsequent steps.

Next, the images captured by the wide-angle mono lens module 110, thenarrow-angle mono lens module 120, and the color lens modules 130A and130B are processed into a composite image. For example, the imagepre-processing module 542 performs image rectification on the main monoimage, the auxiliary mono image, and the two color images to generate amain mono rectified image, an auxiliary mono rectified image, and twocolor rectified images. More specifically, the image pre-processingmodule 542 rectifies errors of the brightness, color, and geometricposition of the four images caused by the wide-angle mono lens module110, the narrow-angle mono lens module 120, and the color lens modules130A and 130B respectively.

In this embodiment, the image pre-processing module 542 receives aplurality of rectification parameters in association with the wide-anglemono lens module 110, the narrow-angle mono lens module 120, and thecolor lens modules 130A and 130B from the storage unit 52. Therectification parameters may include intrinsic parameters and extrinsicparameters of the lens module array 100, which are for performing theimage rectification. The intrinsic parameters indicate a conversionrelationship between camera coordinates of the lens module array 100 andimage coordinates, that is, using a pinhole imaging principle to projectthe camera coordinates of the lens module array 100 to a projectiveplane. For example, the intrinsic parameters include focal length, imagecenter, principal point, and distortion coefficient. The extrinsicparameters indicate a conversion relationship between a world coordinatesystem and a camera coordinate system of the lens module array 100. Forexample, parameters related to a position and an image capturingdirection of the portable device 50 in a three-dimensional coordinatesystem, such as rotation matrix and translation vector, etc. Inaddition, the rectification parameters may also include parametersrelated to illumination compensation or color correction. Nevertheless,the invention is not limited thereto. The image pre-processing module542 performs the image rectification by rectifying the main mono image,the auxiliary mono image, and the two color images based on theaforementioned rectification parameters.

Thereafter, the feature point analysis module 543 performs feature pointdetection on the main mono rectified image and the auxiliary monorectified image, so as to detect an overlapping regions of the main monorectified image and the auxiliary mono rectified image, therebyobtaining a pixel shift characteristic and pixel depth information ofthe overlapping regions. More specifically, the feature point analysismodule 543 detects a plurality of feature points of the main monorectified image and the auxiliary mono rectified image by edgedetection, corner detection, blob detection, or other feature pointdetection algorithms. Thereafter, the feature point analysis module 543finds a group of corresponding feature points of the same characteristicfrom the feature points described above and identifies the overlappingregions in the main mono rectified image and the auxiliary monorectified image respectively. The feature point analysis module 543 cancalculate a depth of field of each pixel in the overlapping regions ofthe wide-angle mono lens module 110 and the narrow-angle mono lensmodule 120 to obtain depth information of the overlapping regions, whichis recorded in the form of a depth map, for example.

Then, zooming and warping is performed based on a command inputted bythe user, that is, a zoom ratio is changed (Step S120). If the zoomratio is between 1 and a main/auxiliary mono image ratio, the imagezooming and warping module 544 performs zooming and warping on the mainmono rectified image, the auxiliary mono rectified image, the two colorrectified images based on the zoom ratio, each pixel shiftcharacteristic, and the pixel depth information respectively, so as togenerate a main mono warped image, an auxiliary mono warped image, andtwo color warped images. Here, the “main and auxiliary mono imagemagnification” refers to the magnification between the main monorectified image and the auxiliary mono rectified image, which is fixedand determined in advance. The zoom ratio is a degree of change of thesize of the main image that the user intends to see or output, and maybe set by the user or may be a preset value of the portable device 50.The image zooming and warping module 544 may perform the zooming andwarping on the main mono rectified image, the auxiliary mono rectifiedimage, and the two color rectified images based on the zoom ratio byusing relative shift, deflection, and depth characteristics of twooverlapping regions respectively, so as to generate an image conformingto the zoom ratio requirement of the user and having an overlappingregion viewing angle identical to the appearance. In addition, a rangeof the image warping is associated with the depth information of the twooverlapping regions.

Following that, the image fusing module 545 performs image fusing on theoverlapping region of the main mono warped image and the auxiliary monowarped image according to the zoom ratio, so as to generate a monodigital zoomed image (Step S130). Specifically, the image fusing module545 sets weights required for fusing the images for the overlappingregions of the main mono warped image and the auxiliary mono warpedimage according to the zoom ratio, which are defined as a “firstweight-set” and a “second weight-set” respectively. Then, the imagefusing module 545 weights a mixing ratio of pixel gray scales of the twooverlapping regions by using the first weight-set and the secondweight-set to perform image fusing on the two overlapping regions. Here,the product generated by the image fusing is defined as a “fusedoverlapping region.” Thereafter, the image fusing module 545 replacesthe overlapping region in the original main mono warped image with thefused overlapping region, so as to generate a high-quality digitalzoomed mono image target.

In another embodiment, when the zoom ratio is less than 1, only the mainmono rectified image and the two color rectified images are used toperform digital zoom-out and subsequent image warping process andgenerate the main mono warped image and the two color warped images, andthe main mono warped image is directly set as the digital zoomed monoimage target. On the other hand, when the zoom ratio is greater than themain/auxiliary mono image magnification, only the auxiliary monorectified image and the two color rectified images are used to performdigital zoom-in and subsequent image warping and generate the auxiliarymono warped image and the two color warped images, and the auxiliarymono warped image is directly set as the digital zoomed mono imagetarget.

Next, the coloring module 546 first performs pixel alignment on the twocolor warped images and the digital zoomed mono image target obtainedthrough fusing respectively. Thereafter, with the digital zoomed monoimage target as the basis, the pixel color at the corresponding positionin the two color warped images is extracted as the color informationthat each mono pixel lacks and fused and filled in the mono pixelposition to generate a composite image, which is a digital zoomed colorimage, for example (Step S140). Finally, the digital zoomed color imageis outputted. For example, the digital zoomed color image obtained bycompleting the steps described above is displayed on a screen (notshown) of the portable device 50 (Step S150).

To conclude the above, the invention utilizes the images captured by thewide-angle mono lens module and the narrow-angle mono lens module as thebasis of the digital zoomed image, thereby significantly improving boththe image resolution and the signal to noise ratio. Since the resolutionis improved, under the condition that the area of the sensing elementremains the same, the thinner narrow-angle mono lens module provides ahigh-magnification optical zoom effect. Using the color informationacquired by the color lens modules allows the coloring module to colorthe digital zoomed image. In addition, the invention disposes two colorlens modules on two sides of the main mono lenses and the auxiliary monolenses, thereby solving or significantly improving the occlusion problemof the conventional dual lens module during coloring of the coloringmodule.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed embodimentswithout departing from the scope or spirit of the invention. In view ofthe foregoing, it is intended that the invention covers modificationsand variations provided that they fall within the scope of the followingclaims and their equivalents.

What is claimed is:
 1. A lens module array for being assembled to aportable device, the lens module array comprising: a wide-angle monolens module; a narrow-angle mono lens module; and two color lensmodules, wherein when the wide-angle mono lens module, the narrow-anglemono lens module, and the two color lens modules are assembled onto theportable device, the wide-angle mono lens module, the narrow-angle monolens module, and the two color lens modules are located at four verticesof a quadrangle respectively, and the two color lens modules are locatedat two opposite vertices.
 2. The lens module array according to claim 1,wherein an angle of view of the wide-angle mono lens module is in arange of 70-80 degrees and an angle of view of the narrow-angle monolens module is in a range of 35-40 degrees.
 3. The lens module arrayaccording to claim 2, wherein an angle of view of the two color lensmodules is in a range of 70-80 degrees.
 4. The lens module arrayaccording to claim 1, wherein the quadrangle is square, rectangular,rhombic, or kite-shaped.
 5. The lens module array according to claim 1,wherein a total number of pixels of the wide-angle mono lens module isthe same as a total number of pixels of the color lens modules.
 6. Animage sensing device, comprising: an image processing module; awide-angle mono lens module; a narrow-angle mono lens module; and twocolor lens modules, wherein when the wide-angle mono lens module, thenarrow-angle mono lens module, and the two color lens modules areassembled onto a portable device, the wide-angle mono lens module, thenarrow-angle mono lens module, and the two color lens modules arelocated at four vertices of a quadrangle respectively and the two colorlens modules are located at two opposite vertices, and the imageprocessing module processes images captured by the wide-angle mono lensmodule, the narrow-angle mono lens module, and the two color lensmodules into a composite image respectively.
 7. The image sensing deviceaccording to claim 6, wherein an angle of view of the wide-angle monolens module is in a range of 70-80 degrees and an angle of view of thenarrow-angle mono lens module is in a range of 35-40 degrees.
 8. Theimage sensing device according to claim 7, wherein an angle of view ofthe two color lens modules is in a range of 70-80 degrees.
 9. The imagesensing device according to claim 6, wherein the quadrangle is square,rectangular, rhombic, or kite-shaped.
 10. The image sensing deviceaccording to claim 6, wherein a total number of pixels of the wide-anglemono lens module is the same as a total number of pixels of the colorlens modules.
 11. The image sensing device according to claim 6, whereinthe image processing module comprises a storage unit and a processingunit, and the storage unit is coupled to the processing unit.
 12. Theimage sensing device according to claim 11, wherein the processing unitcomprises an image inputting module, an image pre-processing module, afeature point analysis module, an image zooming and warping module, animage fusing module, and a coloring module.
 13. A fusing method of adigital zoomed image, the method comprising: capturing images by awide-angle mono lens module, a narrow-angle mono lens module, and twocolor lens modules respectively, wherein the wide-angle mono lensmodule, the narrow-angle mono lens module, and the two color lensmodules are located at four vertices of a quadrangle respectively, andthe two color lens modules are located at two opposite vertices; andprocessing the images captured by the wide-angle mono lens module, thenarrow-angle mono lens module, and the two color lens modules into acomposite image.
 14. The fusing method according to claim 13, wherein amethod of processing the images into the composite image comprises:fusing mono images captured by the wide-angle mono lens module and thenarrow-angle mono lens module into a digital zoomed mono image accordingto a zoom ratio; and fusing the digital zoomed mono image and colorimages captured by the color lens modules into the composite image thatconforms to the zoom ratio and is colorful.
 15. The fusing methodaccording to claim 14, wherein a method of fusing the digital zoomedmono image and the color images captured by the color lens modulescomprises: performing pixel alignment on the digital zoomed mono imageand the color images captured by the color lens modules respectively;and utilizing the digital zoomed mono image as a basis and filling colorinformation in the color images captured by the color lens modules intocorresponding pixels at corresponding positions in the digital zoomedmono image to obtain the composite image.